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INTELLIGENT MODELING. TRANSFORMATION AND MANIPULATION 

SYSTEM 

CROSS REFERENCE TO RELATED UNITED STATES PATENT 

APPLICATION 

This patent application is related to United States provisional patent 
application Serial No. 60/168,020 filed on November 30, 1999 entitled 
INTELLIGENT MODELING. TRANSFORMATION AND MANIPULATION (IMTM) 
SYSTEM. 

FIELD OF THE INVENTION 

The present invention relates to a method of intelligent 2D and 3D object 
and scene modeling, transformation and manipulation and more particularly this 
invention relates to the field of computer modeling, virtual reality, animation and 
3D Web streaming. 

BACKGROUND OF THE INVENTION 

Studies of the human vision system show that the analysis of dynamic 
scene involves both low-level processing at the retina and high-level knowledge 
processing in the brain, see P. Buser and M. Imbert, Vision, translated by R. H. 
Kay, pp. 137.151. The MIT Press. 1992. Su-Shing Chen, Structure form Motion 
without the Rigidity Assumption, Proceedings of the 3rcl Workshop on Computer. 
For motion analysis, it has been shown that the human vision system captures 
both high-level structures and low-level motions of a dynamic scene, see D. Bun* 
and J. Ross. Visual Analysis during Motion, Vision, Brian, and Cooperative 
Computation, pp. 187-207, edited by M. A. Arbib and A. R.Hanson, The MIT 
Press, 1987. Unfortunately, current vision and graphics systems do not satisfy 
this requirement. Popular representations such as those taught by M. Kass, A. 
Witkin and D. Terzopoulos, Snakes: Active Contour Models. International 
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Journal of Computer Vision, pp.321.331 . Kluwer Academic Publishers. Boston, 
1988, and D. N. Lee, The Optical Flow Field: The Foundation of Vision, 
Philosophical Transactions of the Royal Society of London, B290, pp. 169- 
179.1980, do not enable symbolic or knowledge manipulation. These systems 
5 generally lack the capability of automated learning unknown types of motion and 
movements of modeled objects. 

In the fast growing virtual reality society, realistic visual modeling for virtual 
objects has never been so eagerly needed in its history. Previous modeling 
techniques mainly look after geometrical appearance or physical features. 
10 However, as pointed out in J. Bates. Deep Stmcture for Virtual Reality, Technical 
Report, CMU-CS-91-133. Carnegie Mallon University. May. 1991, and G. Burdea 
and P. Coiffet, Virtual Reality Technology, John Wiley and Sons, Inc., 1 994, an 
o ideal object modeling has at least the following requirements: appearance 

modeling for geometric shapes; kinematics modeling for the rotations and 
translations of objects; physical modeling of various properties such as the mass, 
inertia, defonnation factors of objects to mention just a few; and behavioral 
features such as intelligence and emotions. 

Similar requirements arise from many Internet applications where there is 
a fast growing interest in 3D Web contents. Cun-ent 3D Web methodologies 
W 20 (such as VRML, see D. Brutzman. M. Pesce. G. Bell, A. Dam and S. Abiezzi 
p VRML. Prelude and Future. ACM SIGGRAPH '96, pp. 489-490, New Orleans, 

August 1 996) heavily depend on Internet bandwidth to transmit 3D data. 
Therefore, an efficient 3D representation is required to "compress" the complex 
and multi-aspect 3D data. 

To satisfy all these requirements, there is needed a generic stmcture to 
enable symbolic operations for the modeling and manipulation of real and virtual 
world data with different types of information. The representation of visual object 
has preoccupied computer vision and graphics researches for several decades. 
Before the emergence of computer animation, research mainly focused on the 
30 modeling of rigid shapes. Despite the large body of work, most techniques 
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lacked the flexibility to model non-rigid motions. Only after the mid 1980's were a 
few modeling methodologies for solving deformation problems. 

An effective object modeling methodology should characterize the object's 
features under different circumstances in the application scope. In the early 
5 research, focus was placed on appearance modeling since the objects involved 
in vision and graphics applications were mostly simple and stationary. Cun-ently, 
simple unilateral modeling can no longer satisfy the requirement in dynamic 
vision and computer animation. As discussed In G. Budea and P. Coiffet. Virtual 
Reality Technology, John Wiley and Sons, Inc., 1994, a complete 3D object 
10 modeling should ideally comprise at least the following components. 

1) Geometrical modeling: this is the basic requirement for any vision or graphics 
system. It describes an object's geometrical properties, namely, the shape (e.g.. 
polygon, triangle or vertex, etc.) and the appearance (e.g., texture, surface 
reflection and/or color). 

2) Kinematics modeling: this specifies an object's motion behaviors that are vital 
for dynamic vision and animation. 4x4 or 3x4 homogeneous transformation 
matrices can be used to Identify translations, rotations and scaling factors. 

^ 3) Physical modeling: physical modeling is required for complex situations where 

i== the object is elastic and/or defomiations and collisions are involved. Objects can 

g 20 be modeled physically by specifying their mass, weight, inertia, compliance, 

defonnation parameters, etc. These features are integrated with the geometrical 
modeling along with certain physical laws to form a realistic model. 
4) Behavior modeling: this is the least studied aspect of modeling. In intelligent 
modeling, an object can be considered as an intelligent agent that has a degree 
25 of intelligence. It can actively respond to its environments based on its behavioral 
rules. 

Hereinafter, current modeling methodologies are reviewed and classified 
into three categories: continuous modeling, disaete modeling and graph-based 
modeling. 
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Continuous Modeling 

This type of modeling approximates either the whole or a functional part of 
the 3D object by a variation of geometrical primitives, such as blocks, 
polyhedrons, spheres, generalized cylinders or superquadrics. These 
5 geometrical primitives can be expressed as continuous or plecewise continuous 
functions in 3D space. Kinematic and physical features can be easily combined 
with the geometrical shapes. Among the large body of the geometrical primitives, 
generalized cylinders and superquadrics are the popular ones since they could 
easily handle deformations. Barr is considered as one of the first to borrow the 
10 techniques from linear mechanical analysis to approximate visual 3D objects, 
see A. Ban-. Superquadrics and Angle-preserving Transformations, IEEE 
Computer Graphics Applications, 18:21-30, 1981, and A. Barr and A. Witkin, 
Topics in Physically Based Modeling. ACM SIGGRAPH '89. Course Note 30, 
New York, 1989. He defined the angle-preserving transformations on 
1 5 superquadrics. Although the original approach is only for computer graphics, It is 
also useful in vision tasks with fruitful results. As a dynamic extension of 
superquadrics. the deformable superquadrics proposed by D. Terzopoulos and 
D. Metaxas, Dynamic 3D models with local and global deformations: Deformable 
Superquadrics, IEEE Transactions on PAMI, 13(7):703-714, 1991, is a physical 
feature-based approach. It fits complex 3D shapes with a class of dynamic 
models that can deform both globally and locally. The model incorporates the 
global shape parameters of a conventional superellipsold with the local degrees 
of freedom of a spline. The local/global representation simultaneously satisfies 
the requirements of 3D shape reconstmction and 3D recognition. In animation, 
25 the behaviors of the deformable superquadrics are governed by motion 

equations based on physics. In 3D model construction, the model is fitted with 
3D visual information by transfomning the data Into forces and simulating the 
motion equations through time. 
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In animation tasks, it is easy to detect, attach and apply geometrical, 
kinematic and physical parameters to continuously modeled objects. However, it 
is difficult for behavioral features since the model lacks a symbolic structure as 
the base to fit In behavioral languages. Furthemnore. to form any real world 
5 objects, approximation by those pre-defined primitives such as generalized 
cylinders or superquadrics is impossible. 
Discrete Modeling 

A wide variety of computer vision applications involve highly irregular, 
unstructured and dynamic scenes. They are characterized by rapid and non- 
10 uniform variations in spatially irregular feature density and physical properties. It 
is difficult to model such objects from any of the four aspects mentioned before 
with continuous elements. Such difficulty arises from the unpredictable behaviors 
J of the objects. Discrete modeling Is able to approximate the surfaces or volumes 

jjj of this kind of objects by vast patches of very simple primitives, such as polygons 

IS 15 or tetrahedrons. 



Since most graphics applications use polygons as the fundamental 
building block for object description, a polygonal mesh representation of curved 
surfaces Is a natural choice for surface modeling as disclosed in G. Turic, Re- 
Tiling Polygonal Surfaces. Computer Graphics {SIGGRAPH '92), 26(2):55^4. 
|20 1992. Polygonal approximation of sensory data is relatively simple and sampled 
i" surfaces can be approximated to the desired precision, see M. A. Khan and J. M. 

Vance, A Mesh Reduction Approach to Parametric Surface Polygonlzation, 1995 
ASME Design Automation Conference Proceedings, Boston. MA, Sept. 1995. 
Physical and kinematic features can be associated more flexibly with either a 
25 single element (a polygon) or a group of elements (a patch of polygons). 

Triangular mesh is a special case of polygonal mesh. It has been 
recognized as a powerful tool for surface modeling due to its simplicity and 
flexibility and abundance of manipulation algorithms. They are used in many 
general vision and graphics applications, which provide fast preprocessing, data 
30 abstraction and mesh refinement techniques, see see for example R.E. Fayek, 
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3D Surface Modeling Using Topographic Hierarchical Triangular Meshes, Ph.D 
Thesis, Systems Design Eng.. University of Waterloo. Waterloo, Ontario, 
Canada. April 1996, L. De Florlani and E. Puppo, Constrained Delaunay 
Triangulation for ly/lulti-resolutlon Surface Description, Pattern Recognition, pp. 
5 566-569. 1 988. and S. Rippa, Adaptive Approximation by Plecewise Linear 
Polynomials on Triangulations of Subsets of Scattered Data. SIAM Journal on 
Scientific and Statistical Computing, pp. 1123-1141,1992. 

Based on the triangular mesh. Terzopoulos and Waters had successfully 
attached physical constraints on the human facial model as disclosed in D. 
10 Terzopoulos and K. Waters, Analysis and Synthesis of Facial Image Sequences 
Using Physical and Anatomical Models, IEEE Transactions on RAMI, 16(6):569- 
579,1993. From sequences of facial images, they built mesh models with 
g anatomical constraints. An impressive advance of the methodology is that it has 

the capability to model the behavioral features. With the support of anatomical 
Jl 15 data, different emotions can be modeled and applied to arbitrary human faces. 
I "^^^ '"ain drawback of discrete methodologies is that it lacks the high level 

,n staicture to control the modeling or to perform symbolic operations. Furthermore, 

the discrete primitives are unstructured and only contain Information about local 
H features. When applied to dynamical cases, even though the geometrical 

g20 modeling can be highly precise, abstracting high level information from the data 
g is still problematic. 

Graph'Based Symbolic Modeling 

In graph-based approaches, a complex object Is usually represented 
explicitly by a set of primitives and the relations among them in the fomi of a 
25 graph. If the primitives are regular blocks such as cylinders, cubes or 

superquadrics, such model can be viewed as a clone of continuous modeling. If 
the model consists of vast number of primitives that are discrete polygons or 
polygonal meshes, it is an extension of discrete model. 

The graph representation was first put fonward in 1970 neither for 
30 computer vision nor for graphics applications, but for the script description of a 
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scene in artificial Intelligence, see A. C. Shaw. Parsing of Graph-Representation 
Pictures, Journal of ACM, 17:453-481. 1970. In the early 1980's. Shapiro (LG. 
Shapiro, Matching Three-dimensional Objects using a Relational Paradigm, 
Pattern Recognition, 17(4):385-405, 1984) applied relational graphs for object 
representation where the quadtree (for surface) and octree (for solid) encoding 
algorithm given by Meagher (D. J. Meagher, Octree encoding: a new technique 
for the representation, manipulation, and display of arbitrary three-dimensional 
objects by computer, Tectinical Report, IPL-TR-SO, 111, Image Processing 
Laboratory, Rensselaer Polytechnic Institute, Troy, NY. April 1982). It can be 
viewed as a special case since they form trees (directed graphs) as the 
hierarchical structure for representation. In the same period. Wong et al. 
introduced the attributed hypergraph representation for 3D objects see A. K. C. 
Wong and S. W. Lu. Representation of 3D Objects by Attributed Hypergraph for 
Computer Vision, Proceedings of IEEE SMC International Conference, pp. 49- 
53, 1983. Later, random graph (A. K. C. Wong and M. You. Entropy and 
Distance of Random Graphs with Applications to Staictural Pattern Recognition, 
IEEE Transactions on PAMI, 7(6):59g-609. 1985) and more sophisticated 
attributed hypergraphs are presented as geometry and knowledge 
representation for general computer vision tasl<s (A. K. C. Wong and R. Salay, 
An Algorithm for Constellation Matching. Proceedings on of^^ International 
Conf on Pattern Recognition, pp.546-554, Oct. 1986 and A. K. C, Wong. M. 
Rioux and S. W. Lu, Recognition and Shape Synthesis of 3-D Objects Based on 
Attributed Hypergraphs, IEEE Transactions on PAMI, 1 1(3):279-290, 1989. 

In 3D model synthesis, random graphs were applied to describe the 
uncertainties brought by sensors and image processing, see A. K. C. Wong and 
B. A. McArthur, Random Graph Representation for 3-D Object Models, SPIE 
Milestone Series. MS72:229-238, edited by H. Nasr. in Model-Based Vision. 
1991. In Wong et al., an attributed hypergraph model was constojcted based on 
model features (A. K. C.Wong and W. Liu. Hypei^raph Representation for 3-D 
Object Model Synthesis and Scene Interpretation. Proceedings on 2nd 
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Workshop on Sensor Fusion and Environment Modeling (ICAR), Oxford, U.K., 
1991). The representation had a four-level hierarchy that characterizes: a) the 
geometrical model features; b) the characteristic views induced by local 
image/model features, each of which contains a subset of the model features 
visible from a common viewpoint; c) a set of topological equivalent classes of the 
characteristic views; and d) a set of local image features wherein domain 
knowledge could be incorporated into the representation for various forms of 
decision making and reasoning. 

Since graph-based modeling approaches introduce the concepts of 
primitives and their relations, it is straightfonward for constructing a hierarchical 
representation. At lower levels, geometrical, kinematic and physical features can 
be encoded as the primitives and their associated attributes. At higher levels, 
entities such as edges and hyperedges can be used to represent symbolic 
information. In a graph structure, it Is handy to perform symbolic operations. With 
the aid of machine intelligence, domain knowledge can be learned and later 
recalled and processed together with other data. However, traditional graph- 
based methods lack the representational power for dynamic objects. Although 
they have the potential to handle defomiations/transformations, yet up to now, 
they are applied only to rigid object modeling, due to their rigid structure and the 
lack of transformation operators on graphs. 

The fast paced developments in computer vision, computer graphics and 
Internet motivate the need for a new modeling methodology using a unified data 
structure and manipulation processes directed by machine intelligence. 



SUMMARY OF THE INVENTION 

An object of the present invention is to provide a generic and unified 
object modeling for computer vision, computer graphics, animation and 3D 
Internet using machine intelligence. 

In one aspect of the invention there is provided a computerized method of 
modeling a three dimensional object, comprising: 
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extracting selected topological features from captured imaging data or 
other surface data of a three dimensional object; 

constructing a triangular mesh representation of said three dimensional 
object from said topological features; 
5 mapping vertices and edges of said triangular mesh to vertices and edges 

respectively of a representative attributed graph; and 

constructing an attributed hypergraph representation from said 
representative attributed graphs. 

In another aspect of the invention there is provided a computerized 
10 method of Intelligent model transformation, comprising the steps of: 

computing an optimal subgraph Isomorphism for attributed hypergraph 
between a source attributed hypergraph and a target attributed hypergraph. 
wherein said source attributed hypergraph Is to be transformed Into said target 
attributed hypergraph; 

computing a sequence of transformation operators from said optimal 
subgraph Isomorphism for attributed hypergraph; 

computing a transformation path from said sequence of transformation 
operators; and 

generating a sequence of attributed hypergraphs along said 
20 transformation path from said sequence of transformation operators. 

In another aspect of the Invention there is provided a computerized 
method of Intelligent model augmentation of a real scene with a virtual scene Into 
an augmented scene, comprising the steps of: 

constructing an attributed hypergraph representation of said real scene; 
25 constructing an attributed hypergraph representation of said virtual scene; 

computing a sequence of transformation operators between said two 
attributed hypergraphs; 

integrating said two attributed hypergraph representations into a unified 
attributed hypergraph representation using said sequence of transformation 
30 operators; 

constructing an augmented scene from said unified attributed hypergraph 
representation. 



9 



11/30/2000 IS: 57 HILL 8. SCHUMfiCHER ■> DQUELL NO. 460 



Dll 



1=1 



BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will now be described, by way of example only, reference 
being had to the accompanying drawings, in which: 

Figure 1 is a block diagram of an Integrated computer vision and graphics 
5 system using the intelligent modeling, transformation and manipulation method 
according to the present invention; 

Figure 2 shows the relationships among geometrical, physical and graph 
representations in the view of category theory; 

Figure 3 graphically illustrates a functor F=(Fobj. Fmor) that maps category 
10 Gp/7 to category Sef; 

Figure 4 shows the net-like structure for attributed hypergraph with nodes 
and links; 

Figure 5 shows the primary operators for attributed hypergraphs: 5(a) 
union and 5(b) intersection; 



[n P\gwe 6 shows the primary operators for an attributed hypergraph: 

J (a) dichotomy; (b) merge; (c) subdivision; (d) join; (e) attribute transition, 

ij Figure 7 shows the geometrical, physical and behavioral attributes and 

their relations in AHR; 



Figure 8 graphically illustrates a compound operator defined on a simple 

20 graph; 

Figure 9 illustrates examples of continuous transformations with 
qualitative transition operators on simple graphs: (a) subdivision and (b) join; 

Figure 10 graphically illustrates the relationship of spherical coordinates 
and Cartesian coordinates for a three dimensional point; 

Figure 1 1 shows the process to constaict face representation from wire- 
frame model; 

Figure 12 illustrates the process of AHR-based augmented reality 
construction; and 

Figure 13 illustrates the principle of texture inversion from 2D image to 3D 
30 surface. 
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DETAILED DESCRIPTION OF THE INVENTION 



The present invention provides a new forni of attributed hypergraph 
representation (AHR). which when coupled with triangular mesh surface 
approxinfiation. provides a highly efficient method for both 2D and 3D modeling. 
For 3D objects, surface modeling is preferred to solid modeling since it fits both 
computer vision and computer graphics applications as common vision sensors 
provide surface data only. A net-like data structure is designed to handle the 
dynamic changes of the representation corresponding to the objecfs movements 
and deformations so as to overcome the inflexibility of graph structure. Figure 1 
gives a blocl< diagram of the integrated Intelligent modeling, transfonnation and 
manipulation system based on AHR. 

For any scientific study on the representation, in particular, for general 
object modeling, the fundamental problem Is: given a collection of instances of a 
certain entity, a set of transformations that can be applied on the entity, and an ' 
isomorphism or equivalent relation with respect to the transformations, find the 
general description of the entity and the associated transformations in a 
mathematical language. In this invention, categorical formalism is brought upon 
to generalize the above problem, since It provides a mathematical language to 
address objects, transformations, invariance and equivalence which is essential 
for a general pattern representation and manipulation. 

The universality of the categorical language enables formal definitions of 
notions and procedures which. In the past, were empirically defined in the 
computer vision and computer graphics settings. Many concepts in category 
theory can play predominant roles In the formalization of most general 
representation problems. It also helps to establish links between apparently very 
different notions (such as projections and transformations), or between different 
aspects of a same problem (such as geometrical modeling and physical 
modeling). 

I. Definitions and Notations 
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[Definition 1] A category C is given if a class of elements, called objects, are 
given such that: 

1. for each pair of objects (Of. 02) in C. a map u. called a morphism (denoted 
as 01 _2_> 02) is given; 
5 2. for each object O in C. there Is an identity morphism io, such that, if O 
— O; then 0'=0; 

3. the composition (denoted as *) maps of morphisms satisfy the following 
axioms: 

1) if 01 02— 03 04. then 01 > 03 > 04 and 
10 Of _!_> 02 04 

2) identity maps always compose with 0 " > O' to give to' * u^u and w • 
io ~ u. 

M The categories commonly met In mathematics have objects that are structured 

y sets and morphisms. 

^ (Definition 2] A morphism u from 0 to 0'. denoted as 0 - > 0'. is called a 

ifl retraction if there exists a morphism v from O' to O, such that O' " > O and w * 

V = fo'. A morphism u from 0 to O' is called a coretraction. if there exists a 
M morphism vfirom O'to O such that O' _jl_> O and v* w = /o. A morphism u Is an 

p 20 isomorphism, if it is both a retraction and a coretraction. and then O is said to be 
g isomorphic to 0', denoted as 0 - O'. If 0 - 0' and O' ~ O" then O - 0". 

(Definition 3] A covariant functor F from category CI to category 02 is defined 
as a pair of maps F = CFod/ Fmor): 
25 • objects of 01 -fiSL* objects of C2; 

• morphisms of 01 morphisms of 02. 

or simply as Cf-£-» 02. Usually we refer to a covariant functor simply as a 
functor. 
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Figure 3 gives an example of a functor. G1 and G2 are two objects In category 
Gph. and a morphism u is defined such that G1 G2. A covariant functor F= 
rFo6>, Fmor; is defined as the mapping from graph {Gph) to set (Set). We have 
G1 S1 and G2 S2. Corresponding to u. the morphism between S1 
and S2 Is Fmor(u). 



[Definition 4] If a functor F defined on categories C1 and C2 presen/es all 
retractions (coretractions, isomorphisms) from C1 to C2, it is called a retraction 
preserving (coretraction preserving, isomorphism presen/ing) functor. 

[Definition 5] The composition of two functors F1 on categories C1, C2 (CI 

C2) and F2 on categories C2, C3 {C2 C3) is defined as the functor F2 
* F1 on C1 and C3, such that C1 > C3 

[Definition 6] Two categories C1 and C2 are called equivalent if there exists two 
functors F1 and F2, such that C1 C2 and C2 C3. and the 
compositions of the two functors have the properties 

1. F2*FUIci; 

2, F1*F2^IC2. 

With the concept of functor defined within the scope of computer vision and 
graphics applications, the geometrical, physical and graph representations can 
be viewed as different categories. In 3D case, their relations are shown In Figure 
2. The graph-based representation can be viewed as the result of staictural 
abstractions from geometrical and physical representations, by applying the 
functors that map Geo and Phyio Gph respectively. 
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(Definition 7] A hypergraph is an ordered pair G = (X, Y), where X = {w 
n) is a finite set of vertices and Y - {Hj ( 1^J<, m) is a finite set of hyperedges; 
each HJ is a subset of X such that Hi uH2 <j...uHmsX. 



5 [Definition 8] An attributed hypergraph (AH) G is a hypergraph defined as an 
ordered pair G=(X, Y) associated with AXan6 AY, where G =(X, V9 is a 
hypergraph. AJCis a finite set of vertex attribute values and >Ay is a finite set of 
hyperedge attribute values, vi in X may be assigned values of attribute from AX. 
and Hj in Y may be assigned values of attribute from AY. 

t^e present invention, the hypergraph is implemented with a net-like 
dynamic data structure (an example in Figure 4). The term "dynamic" signifies 
that the data structure supports structural operations such as join and 
subdivision on hypergraphs on the fly, which. In the view of category theory, are 
related to certain changes of the object via functors. 
^ a dynamic structured hypergraph, the basic element is called a node 

m Shown as a rectangle. The nodes at the bottom level represent the vertices; the 

f nodes at the intermediate layers are the hyperedges which are supersets of the 

H nodes at their lower levels; and the node at the top layer (called the root) 

represent the entire hypergraph. 
5 With such a data structure, the structural changes on hypergraphs can be 

g handled efficiently. For example, the join of two hyperedges is simply the 

combination of the two con-esponding nodes and the re-organization of the 
associated links. Graph-based computations such as the re-organization of a 
hyperedge based on certain criteria can be performed by reshuffling the links 
25 between the nodes. However, In traditional hypergraph data representations 
such as the incidence matrix or adjacency matrix, any topological change could 
initiate a re-organlzation over many entities. Data elements with or without 
relations to the nodes to be operated on may have to be accessed. 

In a hypergraph, information represented by nodes at higher layers is 
30 abstracted from those at lower layers. The nodes at higher layers are important 
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for intelligent manipulation and knowledge processing, while those at lower 
layers are crucial for complete and accurate modeling. The nodes at the bottom 
layer construct an elementary graph. 



10 



[Definition 9] In a dynamic stmctured attributed hypergraph, the nodes at the 
bottom layer are called the elementary nodes, the linl<s among the elementary 
nodes are called the elementary edges. The attributed graph, which consists of 
the elementary nodes and the elementary edges, is called the elementary graph. 

If a hypergraph has n layers, and the node set at layer / is X/, then 
hypergraph G = (X, V9 can be written In the form of G = Xi. . . . , Xn) where 
X = and y = {X2, . . ., Xn). The meanings of the nodes at the hyperedge 
layers nomially depend on the applications. In most applications, we prefer that 
there Is a layer of hyperedges that characterizes the object's higher level 
properties. For example, In the task of environment modeling for indoor 
J| 1 5 navigation, we can have the hyperedge layers on top of the elementary graph: 
jS • The elementary graph characterizes the basic features captured from the 

sensors such as comers and lines; 

• The hyperedges at the hyperedge layer extract the organization of the bottom 
nodes, representing higher level knowledge such as the layout of the furniture 

I 20 in a room or the organization of the rooms in a building; 

• The root node represents the entire scene. 



--4 
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The bottom layer of an attributed hypergraph should simultaneously satisfy 
the following two requirements: 
25 • The ability to represent subtle features for general patterns, from which one 

can reconstruct a complete object; 
• The suitability to integrate the representation into a stmctural frameworic. 



30 



The use of the mesh-based representation is ideal due to the unstnjctured 
nature of general 3D shapes. Compared with other mesh-based methodologies 
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such as NURBS, the triangular mesh has the advantages that it Is simple and 
flexible. In the vertex-edge structure, a triangular mesh can be written in the form 
of 7= (Vt, Et) where Vt is the vertex set and Et Is the edge set of the mesh. 

5 [Definition 1 0J A representative attributed graph (RAG) G= {V, E) of a triangular 
mesh 7 is an attributed graph constructed In the following manner 

• for each vertex vt e W, there is a vertex vaeV corresponding to it. denoted 
as: w -> va\ 

• for each edge et e Et, there Is an edge ea e E corresponding to it. denoted 
10 as;er->ea; 

• vfs features are mapped to wa's features; 

• ef's features are mapped to es's features. 

The AHR of a triangular mesh, which represents a 3D shape, is based on 
j 15 such a RAG. The elementary nodes of an attributed hypergraph (i.e., the nodes 
at the bottom level) consist of the vertices in the RAG, and the elementary edges 
are copied from the edges in the RAG. The properties of the surface attached to 
the triangular mesh, such as location, orientation, area, color, texture, mass 
density and elasticity, are mapped to the attributes associated with the vertices 
1 20 or edges. Thus, the RAG of the mesh directly constructs the elementary graph of 
O the AHR. 

IDefinitlon 11] Suppose that X = {xi. x2, . . ., xn} is the vertex set of attributed 
hypergraph H = (X. Y), E ^{xei, xe2, . . ., xsm) (E eX. ek <n;k^ 1,2. . . . . m), 
25 and asi, ae2, .... ae/n are the vertex attributes associating with vertices xei, xe2. 
....xem respectively. We say that 6 is a hyperedge induced by attribute value a, 
if for a selected attribute value a, we have: 

• a e {ak\ 1^k<n} If the corresponding vertex xk e E: 

• a <s {ak\ H n} if the corresponding vertex xk « E; 



16 



11/30/2000 16:57 HILL & SCHUMfiCHER ^ DOUIELL NO.4G0 

• e, as a subgraph of H, is connected. 

[Definition 12] The union of two hypergraphs Gi==(Xi. Yi) and G2= (X2. Y2), is to 
form a new hypergraph G = (X, Y). denoted as G = Gf u G2. such that X = Xi\j 
5 X2 and Y=Yiu Y2. 




[Definition 131 The intersection of two hypergraphs G1 = (Xi, Yi) and G2 =(X2, 
Y2), is a new hypergraph G = (X, V;, denoted as G = Gir> G2, such that X = Xfr. 
X2and y=yfn Y2. 

Figure 5(a) illustrates the operations of union, and Figure 5(b) illustrates 
intersection on two simple hypergraphs. 

[Definition 14) The dichotomy of a vertex vd2&X in hypergraph G-(X.Y), 
denoted as (vai 0 vd2) = vd, can be obtained by the following steps: 

1 . in X, replace vd by two vertices vdl, vd2\ 

2. in Y, replace w in all hyperedges by two vertices vdi, vd2\ and 

3. add a new hyperedge Hd to V where Hd = {vdi, vd2}. 
An example of dichotomy is shown in Figure 6 (a). 

[Definition 15] If in hypergraph G -(X, Y), two vertices we Xand vj e Xare 
adjacent in n hyperedges HI1. Hi2. .... Hm 'mY(n^ 1). then the merge of vi and vj, 
denoted as vm = vi ® vj, can be obtained by the following steps: 

1 . in X, replace vi and vj by a single vertex vm ; 

2. in H/y, replace viand vj by a single vertex vm; 

3. replace vi or v^? by vw? in any other hyperedges in Y ; 

4. if a hyperedge H contains only vi or vj, it is nullified. 
An example of merge is shown in Figure 6 (b). 
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[Definition 16] The subdivision of hyperedge Hs - {vli, vi2, . . vin} In 
hypergraph G^(X,Y) (where vll € X. i ^ 1, 2. . . n and q Y), denoted as 
(H&1, Hs2, Hsn) = a Hs. can be obtained by the following steps: 
1 . add a new vertex w© to X, 

5 2. add n hyperedges Hsi, Hs2. .... Hsn to Y. such that Hsi = (vii, ve} (1< i £ n); 
3. nullify Hs. 

Figure 6 (c) Illustrates an example of subdivision on a simple hypergraph. 

[Definition 17] If in hypergraph G = (X. V), HJ1 e V and Hj2 e Y are two 
10 hyperedges with at least one common vertex, then the Join of Hjl and Hj2 in 

hypergraph G-(X.Y), denoted as Hj = Hjl v Hj2. is to: 
^ 1 . add a new hyperedge Hj e Y such that Hj =Hjl u Hj2; 

5 2. nullify the common vertices of Hji and Hj2 in Hj; 

^ 3. nullify and /^2. 

W 15 An example ofjoin is shown in Figure 6(d). 

?Si 

i « 

[Definition 18] The attribute transition of a hyperedge H e V (or vertex «/ e X) in 
H an attributed hypergraph G = (X,Y) denoted as >^>y= fa(>\HD (or >\V = fa(Av)), is a 

y mapping AH-^ A'H (or yAv /A V) which transfbnns the attribute value AH 

g 20 (or attribute value Av) without any change to Hs connectivities (refer to Figure 6 
5 (e)). 



If opt and op2 are two primary operators given In Oefinitions 12 to 18. the 
compound operator © of opi and op2 is defined as applying op2 on the result of 
applying opi, denoted as © = op2 * opi. 

Figure 6 shows an example. The compound operator © given by the figure can 
be written in the form of: (Ho. H7, He) -Hi© H2. In most transformations on an 
AH, a qualitative change operator (union, intersection, dichotomy, merge. 
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subdivision or join) is mostly associated with one or more quantitative change 
operators (attribute transition). 

In the view of category theory, the primary operators and their 
compositions constitute the morphisms between different AH's in the AH 
category (Ahp). Through the mapping of functors, they con-espond to the 
motions of the 3D patterns. 



II. Detailed Description of the Methods of Model Construction. 
Transformation and Manipulation 

10 

A. Intelligent 3D Model Construction 

Figure 1 is a block diagram of an integrated computer vision and graphics 
_^ system using the intelligent modeling, transfomnation and manipulation method 

according to the present invention. The steps in broken boxes are optional and 
1 5 are not directly essential for the method of modeling, transformation and 
ilj manipulation. Of the optional steps, the entire process may be earned out on one 

ry computer, one or more local computers connected together or the internet may 

be used as a transmission medium for working the invention between server 
H computers and client computers. This will be more fully discussed later, 

y 20 The input data of this model construction algorithm can be from CCD 

Q cameras, laser range finders, radar ranger finders, sonar, or any other devices 

Q that generate images. All mentioned imaging devices produce images that 

typically contain thousands, if not hundreds of thousands of image pixels. It is 

neither practical nor productive to cany on object modeling directly at the pixel 
25 level. Thus, the first step of the model construction is to extract the topological 

features from captured imaging data of the three dimensional object shown in 

the image(s). 

The feature extraction process applied In this patent is from the research 
result of Q Gao and A, K. C. Wong ("A Curve Detection Approach on Based on 
30 Perceptual Organization", Pattern Recognition, Vol 26, No. 7, pp. 1039-1046. 
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August 1993), which is based on the relaxation edge-tracing method ("An 
Applications of Relaxation Labeling to Line and Curve Enhancement", S. W. 
Zuker. R. A. Hummel and A. Rosenfeld, IEEE Transactions on Computers, Vol 
26, pp. 394^03, 1977). It has four steps: 1) feature detection: 2) feature 
segmentation; 3) feature classification; and 4) feature grouping. The first step of 
the method detects the salient features such as. for example, points, lines and 
contours from the pixel image, and the second steps finds the topological 
relations among them. In the case of using radar or laser range finders, these 
topological features already have the range information (the distance of the 
feature to the Image projection plane). In the case of using conventional camera, 
to recover the range infomriation, at least two cameras are required for each 
view. Then the range information can be calculated using the method given by T. 
S. Huang ("Determining Three-Dimensional Motion and Structure from Two 
Perspective Views", in Handbook of Pattern Recognition and Image Processing, 
Chapter 14, Academic Press, 1986) or by H. C. Longuet-Higgins ("A Computer 
Algorithm for Reconstructing a Scene from Two Projections", Nature, Vol 293, 
pp. 133-135. 1981). 

The process to create the triangular meshes can be run directly on the 
extracted topological features. However, it is desirable to organize them into a 
geometrical model to represent the shape of the object first, which can simplify 
the process of triangular meshes generation, especially when the features 
extracted are sparse. 

The most stralghtforvvard method to represent a shape is the use of wire- 
frames consisting of the surface discontinuity of the object. The two types of 
elements in a wire-frame model are lines and nodes. The lines correspond to the 
straight or curves, and the nodes are point features such as corners or points of 
abrupt curvature change. We consider the face modeling as a more generic dual 
of the wire*frame model. In a face model, the elements are the faces (surfaces) 
bounded by closed circuits of wire-frame edges. Using faces to model an object 
has several advantages over modeling with the wire-frames: 
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• fewer faces than points/lines are required to describe the same object; 

• a face description enables information on the object, such as geometry, 
surface colors and textures to be attached to the model more easily; 

• a face can be bounded by any type of closed curves in addition to the wire- 
frame edges, making it straightforward to enclose any type of contours in the 
object. 

The procedure to extract faces is illustrated in Figure 11. First, a procedure 
called Tabtet_Point_UneO handles the points, lines and contours. An incidence 
table Is generated as shown in Table 1. where Pi (M,2...,k) are the point 
features, and Lj Q-l,2,...m) are the line or contour features. Empty columns, i.e., 
the lonely points, are considered as noise and are removed from the table. 

Table 1: The incidence table of lines/contours and points. 





Pi 


Pi . 


.. Pk 


Li 


1 


0 


0 


Lm 


0 


0 


1 



Merge_PointsO and MergeJUnesQ are the procedures to eliminate duplicated 
features resulting from potentially noisy input data (e.g., the reflections around a 
sharp edge). Merge_Points() scans through the set of comers. The Euclidean 
distance between pairs of neighboring corners is evaluated. If its value is less 
than a preset threshold (normally the distance threshold is about 2-3 pixels in 
length), the two points are merged Into one. MergeJUnesQ scans through the set 
of lines and contours. If two lines or contours are close to each other and the 
angle between them is very close to 0 orTi. they are merged into one. According 
to the merging result, the incidence table is adjusted. Then the faces are 
extracted by procedure Extract_FacesO. Each face is represented by: 

1 . a set of lines or contours to form the face boundary; 

2. a set of points; 
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3. a vector signifies the normal direction of the face. 

The incidence matrix shown in Table 1 is actually a presentation of a 
connected graph. Each face of the object corresponds to a closed circuit. 
5 Searching for a closed cycle In a graph is a common task in linear graph theory. 
Here a depth-first search algorithm is applied. Different from the conventional 
cycle searching algorithm, in the face searching, there is an constraint to judge if 
a cycle consists of a face or not: in the cycle, all features have to be 
appfxjximately on the same plane. The search algorithm begins with a set of 
1 0 three connected corners, from which an approximated normal vector of the 

possible face can be calculated. During the expansion of the cycle, this nonnal 
vector is consistently updated according to the new topological features added. 
The face grouping procedure includes the following steps: 
1 . set the face set Sf empty; 
fu 15 2. find all ordered lists of 3 corners in which the 3 comers are connected 

consequently by 2 edges, denoted as the list Li = {ci1. ci2, c/3) with comers 
U ci1, ci2 and c/3. where H i1, i2, i3< n, n is the total number of comers, 1^ i ^ 

7 l< 'S the index of the list, and k Is the total number of 3-corner lists; 

3. Iet/=t; 

UJ20 4. for each list Li, calculate the normal vector of the plane defined by all the 
comer points In the list, denoted as ni\ 
6. if the tail of Li is connected to its head, then list Li represents a face; put U in 

Sf, and go to step 8; 
6. if there exists a comer cj in the comer set that satisfies: 
25 • cj Is not In Li yet; 

cj connects the cun-ent tail of Li; 

cj is approximately in the plane defined by all the corners in Li; 
then put cj in Li as the new tail; if no comer satisfies the above conditions, go 
to step 8; 
30 7. go to step 4; 



t 
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8. let / - / -^t . then if / < then go to step 4; 

9. output Sf and exit. 

Finally, procedure Group_Faces() groups the faces together if they have one or 
5 more common edges and their normal vector are collinear/parallel. 

Building a triangular mesh is straightfoHA/ard from the face model. The faces 
are first converted into the forms of polygons or polygonalized circles. The 
following recursive procedure decomposes a polygon PO v\/ith n edges and n 
10 comers into n-2 triangles: 

1. set the triangle list 7 empty, let m = n; 

2. lety = 0; 

3. in polygon P/ start from any point, label the corners from 1 to m 
consequently, denoted as c1, c2, cm, where m is the number of comers in 

fy 15 polygon/^ 
S 4, Iet/ = ^; 

5. add an edge from corner / to comer /+2 to make up a new triangle t = {ci, 
c/>t, C/+2}. and add f to 7(if />2 -m^l, then use c1 as c/>2); 
n 6. let/ = /^2; 

W 20 7. if / < m, go to step 5; 

5 8. use the nodes c1, c3, c2k*1 and the newly added edges to construct 

a new polygon Pj+1 with reduced numbers of comers and edges; 

9, If is a triangle, then add Pj*1 to 7. output 7 and exit; 

10. let j -j +1, go to step 2. 

25 

Since the triangles generated by the above processes are directly from the 
face models, they automatically satisfy the conditions required for a triangular 
mesh. If the topological features extracted from the Imaging data are dense, then 
triangular meshes can be generated using an improved version of the weli- 
30 known Delaunay triangulation method given by R. E. Fayek and A. K. Wong 
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("Triangular Mesh Model for Natural Terrain", Processings on SPIE: hielligent 
Robots and Computer Vision, Oct 1994). However, to apply such method, a 
transformation of the features Is required since Delaunay triangulation only works 
on data that lay approximately on the same plane. The topological features of an 
object are transformed from the conventional Cartesian coordinate system, 
where location is defined by (x, y, z), into a spherical coordinate system centered 
within the object, where location is defined by (a, p. r) (refer to Figure 10). The 
entire process of applying Delaunay triangulation is: 

1 . compute a spherical coordinate system for the three dimensional object to be 



1) compute the geometrical center Pc = (Xc, Yc, Zc) from the coordinates of 
the points extracted from the image: 

Xc = (Xmax-^Xmln)/2.0, Yc = (Yma)(+Ymin)/2.0. Zc = (Zmax-^Zmin)/2.0 
where Xmax, Ymax, Zmax, Xmin, Ymin and Zmin give the range of the 
object occupies in the space expressed In Cartesian coordinate system. 

2) compute the center of gravity Pg = (Xg, Yg, Zg) of all topological features: 



where n is the total number of feature points and (Xi, Yi. Zi) is the 
coordinate of z"* point expressed in Cartesian coordinate system. 

3) compute the major axis, which is given by vector Va = (Pc. Pg); 

4) compute a minor axis, which is given by vector Vm, where Vm is: 



where X signifies the cross product, and Pk is the feature point that has 
the largest distance to vector Va. 
5) construct the spherical coordinate system, such that Pg is the center, Va 
is the direction of a=0, Vi is the direction of M. r is the distance to the 
center. 

2. transform the coordinates of said face model from the conventional three 



modeled; 



Xg^(LXi)/n, Vg = r£V/;/n. 



Zg = (LZiyn 



Vi = Va X (Pc, Pk), 
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dimensional Cartesian coordinate system to said spherical coordinate 
system; for transformed point (a, p, r), its relation to (x. y, z) satisfies: 
X = r sina cosp, y = r s/na s/op, z = r cosp 

3. apply Delaunay triangulation using r=0 as its reference plan, that is, 

5 performing the triangulation on (a, P), while using r as the reference feature. 

4. transform the resulting triangular meshes from spherical coordinate system to 
conventional Cartesian coordinate system. 

After a set of triangular meshes are obtained, they can be directly mapped 
10 to the representative attributed graph (RAG) according to the procedures given 
in Definition 10. Hyperedges are then Induced from the elementary graph in 
hypergraph, which is a dual of RAG. Different from an edge in an attributed 
5 graph, a hyperedge in an attributed hypergraph represents the n-ary relation 

^ among a number of (at least two) vertices, groups of vertices or other 

|r|lS hyperedges in the attributed hypergraph. In this invention, the hyperedges are 
Si Induced from the triangles in the RAG. The steps are: 

1 . select at least a type of attribute used for hyperedge Inducing (for object 
modeling, typical feature types are surface adjacency, orientation, color and 
texture); 

q20 2. select the value bins Bi=[bii, bi2). \=1. 2, ...k,o1 the attribute values for the 
selected attribute type(s). where k is the total number of bins; 
3. group all attributed vertices into groups, such that all attributed vertices In the 
same group and connected and have attributed values fall Into the threshold: 
1 ) set the group list G empty, let n=0; 
25 2) letn=/i+f,y=0: 

3) find a triangle T In the RAG that Is not in any group In G={Gi, G2, ... Gn- 
1), if all triangles have been put into G, then exit; 

4) compute attribute value b of T, find the bin 8/ such that b s Bi, and put T 
in Gn, let y*f ; 
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5) if a triangle Fin the RAG is adjacent to any triangles In Gn, compute its 
attribute value b\ if b'e Q\, then add T into Gn. and let i-l^V, 

6) repeat step 4) until there is no T found that satisfies the condition to be 
put into Gn; 

5 7) go to step 2). 

4. connpute an attribute value by computing the average attributed for all 
triangles in the group and assigned It as the attribute of the group; 

5. assign an attributed hyperedge for each group. 

10 The hyperedge grouping criteria depend on the types of attributes selected 

for the hyperedges. Such attributes can be kinematic attributes, geometrical 
attributes, physical attributes, and/or behavioral attributes. Figure 7 shows the 
levels of abstraction and their relations with triangular mesh and attributed 
hypergraph. 

15 In a typical application, there will be more Image views. For example, for a 

complete camera-based 3D object model, at least four views are preferably 
taken, one for the front, one for the back, and two for the left and the right sides 
of the object. For each view, following the procedure of attributed hypergraph 
constnjction given above, we can obtain one attributed hypergraph. Therefore, a 
Q 20 method to synthesize all these attributed hypergraphs is required to compute a 
complete attributed hypergraph representation (AHR) of the object. The 
procedure comprises the following steps: 

1 . determine the overiapping region of all available attributed hypergraphs Hf, 
H2, ...H<r, where k is the number of attributed hypergraphs: 
25 1 ) transform the coordinates of points In the form of {x, y, z) that correspond 

to the vertices in the hypergraphs to points In spherical coordinate system 
in the form of ^a, p, r) as described before; 

2) for all point ^a, p, r), let x'=a, y'=p and z'=r; 

3) project (o('. y' zO to the plane z'=0; 
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4) determine the overlapping regions Of, 02, ...Om. and non-overlapping 
regions Om*i, ... Om^-n, by examining all projected points. 

2. In eacli overlapped region O/, 7 < / < m-fn, using all the points, re-compute a 
triangular mesh Ti by Delaunay triangulation method as described before; 

3. for triangular mesh Ti, mapping the re-computed triangular meshes to a new 
RAGR/; 

4. form a new attributed hypergraph H'i from each Ti 

5. integrate all the attributed hypergraphs in a new integrated attributed 
hypergraph H by applying the union operater "u" given by Definition 12: 



6. apply the hyperedge inducing method described above to compute attributed 
hyperedges for the new integrated attributed hypergraph H\ 

7. associate the newly computed attributed hyperedge with H to form an 
attributed hypergraph representation (AHR). 

During the synthesis process described above, a more complete AHR Is 
constructed. The possible redundant Infomiation in the input AHR's obtained 
from different views enables one to perform the following improvements on the 
recovered 3D data: 

• Compensate for system errors'. 

System enx>rs are brought in by the digitization error, feature detection enror 
and computation en-or etc. The errors can be compensated by the averaging 
effect in the synthesis. 

• Recover missing features: 

Occlusions, unrealistic illuminations, or en-ors in feature detection can cause 
missing object parts in one view. However, the missing parts still have a good 
chance of being present in other views. 

• Eliminate erroneous features: 



H-H'l^ H'2 u H'm u H'm*1 u H'm*2 u . ..u H'm*n 
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The noises on the object's surfaces and in the background may form 
erroneous features. In AHR synthesis, a score representing the number of 
appearances for each feature can be added. Since the noise has very little 
chance of being repeated at the same place every time, It will have a very low 
5 score. At the final stage, features with low scores can be eliminated. 
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The same model construction concept can be easily applied to 2D 
objects. If the topological features extracted for AHR construction are in 2D 
domain instead of 3D domain, we can easily construct the AHR in the same way. 

10 

6. tnteIHgent 3D Model Transformation 

One of the most important applications of attributed hypergraph 
y representation (AHR) based object modeling is automatic transformation of 3D 

-J objects. A general transformation is basically the metamorphosis of an object 

j Jl 5 from one state (the source) to another (the target) via continuous transitions 
« (morphing). AHR provides an ideal tool for 3D transformation on object's shapes, 

;| colors, textures and many other properties. Given two AHR's representing the 

j\ two states of an object, a set of operators defined for attributed hypergraph can 

I— 

H be extracted. Moreover, a sequence of intermediate 3D objects under continuous 

q20 defonnation or transformation can be reconstructed via the construction of the 

Q intermediate AHR's. 

o 

~ When two states of the same object are represented by AHR's. the 

transformation corresponds to a sequence of operations applied on the source 
AHR such that it finally changes itself into an AHR that Is isomorphic to the target 
25 AHR. The AHR-based transformation disclosed herein is able to: 

1 . Automatically determine a transformation path between two objects; and 

2. Justify the qualitative and the quantitative changes in the transformation. 
To produce the metamorphosis of an object from one state to another one, 

there may exist many paths. There are usually certain constraints controlling the 
30 transformation thus making one path more preferable than others. The volume 
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preserving and the surface preservif)g constraints are the popular ones in many 
applications. These methods usually use Minkowski sums or a merged topology 
to control the time-varying object. An AHR-based transformation is different in 
that it uses feature presen/ing constraints. 
5 In an AHR-based transformation, given the initial AHR Gi and the final AHR 

G2, an optimal subgraph isomorphism algorithm first finds the optimal common 
sub-hypergraph of the two AHR's. Since the distance measure applied In the 
optimal subgraph isomorphism algorithm is confined by the type of selected 
features, the transformation path generated follows the path with the minimal 
10 cost subject to the feature type. In other words, the algorithm attempts to find the 
path that minimizes the cost of subgraph isomorphism associated with the 
selected feature type. Thus. It Is called feature preserving transformation. 

□ The operator sequence that performs the transformations between the two 
Si AHR's can be similarly detemnined. The operator extraction begins with the 
5^5 global motion parameters rotation matrix R and translation vector ffrom all the 
^ matched elements of the two AHR's, which are vertices, edges or hyperedges. 

The least squares method used in estimating R and T minimizes the total 
=^ displacement for the matched elements. Since there are only quantitative 

M changes for those matched ones, the qualitative Information associated with 

p20 them is well preserved. Meanwhile, the optimal subgraph isomorphism algorithm 

□ minimizes the number of unmatched elements since they will Introduce penalties 
~ that have the highest cost in matching. Hence, during the process of deforming 

Gi into G2 by applying the operator sequence, the qualitative features In the 
source AHR are maximally preserved. 

25 If the two AHR's used for transfonmation are constmcted through the 

procedures described in Section A above, they should have four layers each, 
one for root, one for hyperedges, one for edges and one for vertices. If one or 
both have more than one layer of hyperedges, these layers have to be combined 
into one using the hyperedge join operator given by Deflnition 17. The 

30 procedures of AHR-based object transformation are: 
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1. letM; 

2. for same i*^ layer of the two AHR's G1 and 62. by the definition of attributed 
hypergraph (Definition 8 and the description following that, as well as Figure 
4), the layers themselves are two attributed graphs respectively; label them 

5 as Gil and G2i respectively: 

3. for each pair Gi/and G2/. compute the optimal subgraph Isomorphism S/for 
attributed graph as given by A. K. C. Wong. M. L. You and S. C. Chan ("An 
Algorithm for Graph Optimal Monomorphism". IEEE Transctions on S. M. C, 
Vol. 20(3). pp. 628-636, 1990); the distance associated with S/ Is denoted as 

10 di; 

4. if i<4, let and go to step 2; 

5. sum up rf/for M, 2, 3, 4, and let a sequence S'(Si, $2, S3, $4) as the 
3 optimal subgraph isomorphism for attributed hypergraph; 

y 6. compute the transformation operators conresponding to matched graph 

j|l 5 elements (attributed vertices, attributed edges or attributed hyperedges) from 

i S: 

J 1) from S, find all the matched graph elements In the form a common 

^ , attributed subgraph g; 

1=4 2) compute the estimated rotation matrix R and the translation Tfrom the 

q20 displacements of all matched elements; Theoretically, three matched 

y pairs of elements are enough to recover R and T since R and Tonly have 

six degree of freedom plus a scaling'(zooming) factor however, It is 
desired to have more than four pairs and use a least squares method to 
insure the best estimation; 
25 3) back project the calculated R and Tto the elements in the initial graph to 

obtain their computed displacement; 
4) convert each element's displacement into an attribute transition operator 
according to Definition 18; 
7. compute the transformation operators corresponding to unmatched graph 
30 elements from S; 
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1) traverse all elements in g computed in step 6-1) and search for 
unmatched ones that are adjacent to at least one element In g; 

2) if an unmatched node nu is adjacent to node ng in g while nu is in G2 , 
then 

a) apply operator subdivision or dichotomy to ng to yield two new 
elements ni and nz; 

b) apply two attributed transition operators to ni and m respectively to 
transform ni's attribute value to ng's. and n2 to nu& 

3) if an unmatched node nu is adjacent to node ng in g while nu is in G1 , 
then 

a) apply operator join or merge to ng and nu to yield one new element np; 

b) apply an attributed transition operator to np to tranform op's attribute 
value to ngs: 

4) go to step 1) with updated (operators have been applied) Gt, G2, and g 
until there is unmatched element; 

8. enlisting all said transformation operators Into a sequence applied and 
eliminate the ones that cancel each other. 

With the sequence of operators extracted, the transformation path is well 
defined by applying the operators one by one. The automatic transformation 
frame generation is based on the definition of distance between attributed 
hypergraphs given by A. K. C. Wong. M. L. You and S. C. Chan ("An Algorithm 
for Graph Optimal Monomorphism", IEEE Transctions on S. M. C. Vol. 20(3), pp. 
628-636, 1990). If d(Gi. G2) is the distance between two AHR's Gi and G2. we 
can use linear interpolation to generate continuous frames between G1 and G2. 
Suppose that in the transformation, the total time from pattern Gi to G2 is T. 
Given any time t such that 0^t<T, the corresponding transformed AHR at t, 
denoted as G(t) can be expressed as: 



Where G2 -Gi signifies the difference between G1 and G2 measured by the AH 



G(t) =Gl-*-[(tn)* (G2'G1)] 



(1) 
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operator set, and "•" means using its left-hand-side as the arithmetic operator to 
quantify its right-hand-side operator set. 

The connotation of Equation (1) is that can be built by applying a proper 
set of AH operators that varies with i. Suppose that the set of operators to 
5 transform Gi to G2 is Op = fopr, op2, . . opn), where op/ (7 < / < n) can be any 
one of the primary attributed hypergraph operators defined previously 
(Definitions 12 to 18). and the set of operators that transfonn Gi to G{i) is 
expressed as Op(t)={opi(t), op2(t), . . ., opn(t)}. Referring to Equation (1). opi(t) 
Is derived from opi by: 

10 • If opi is an attribute transition operator. opi(t) is also an attribute transition 
operator: 

> opi(t)-(1-VT)opi (2) 

• if opi is intersection, dichotomy or subdivision, then: 

> opi(t) = opi {fO<t<T 
iii5 > opi(t)^lop othenvise (3) 

where lop denotes the identity operator that does not change the input AHR; 
if opt is accompanied by an attribute transition operator fa-qpi, then: 

> fa-opi(t) = (1- t/T)fa-opi (4) 

• If opi is union, merge or join, then: 

> opi(t) = lop \f 0<t<T 

> opi(t) = opi othenivise (5) 
where lop denotes the identity operator that does not change anything; if opi 
Is accompanied by an attribute transition operator fa'Opi, we have: 

> fa-opiit)^ (1- i/T) fa-opl (6) 



=-1 



ry 



25 



Figure 9 gives two examples of continuous transformations on simple graphs 
with qualitative transition operators applied. 
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C. Augemeted Reality by Intelligent 3D Model Manipulation 

This invention provides an efficient and practical methodology for vision- 
based object modeling (Section A) and transfonriation (Section 8) which has 
broad applications in computer vision, graphics and virtual reality. One useful 
5 application is to create augmented reality (AR) by Intelligent model manipulation. 
Augmented reality is a technology that first employs computer vision 
methodologies to estimate the 3D geometry of real world scenes, and then 
facilitates computer graphics technologies to augment the synthesized models 
and to present them in a way similar to that of virtual reality (VR). 
10 To build an augmented scene, virtual objects/scenes have to be combined 

with the real objects/scenes constructed by the process described in Section A. 
With this invention, imaging data from the real world can be sensed and modeled 
Q by AHR. Similarly, virtual scenes are also modeled by AHR. The AHR's from 

::i both the real scene and the virtual scene are then integrated into a unified AHR, 

rw i 5 which represents the augmented scenes. 

m The term augmentation used herein means not only integrating different 

j j AHR's. but also supplementing the AHR's with elements that may appear neither 

s in the real scenes nor In the virtual scenes. With AHR, object properties such as 

L surface colors and textures are represented as attributes, which makes it easy 

^ 20 for us to perfomri the integrating and supplementing tasl^s required by most AR 
g applications. Figure 12 gives a simple illustration on how AHR can be applied in 

"■■^ the construction of augmented reality. 

The procedure of constnjcting augmented reality from a real scene and a 
virtual scene is: 

25 1 . constmcting the AHR of the real scene using the method given by Section A; 

2. constnjcting the AHR of the virtual scene using a computer graphics tool; 

3. computing the sequence of transformation operators between said two 
attributed hypergraphs using the method given in Section 8; 

4. integrating the two AHR's conresponding to the real scene and the virtual 
30 scene respectively into a unified AHR that represents the augmented scene 
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by applying the sequence of transfomiatlon operators computed in step 3; 
and 

6. recontructing thie augmented scene from the AHR obtained in step 4. 
The core of this procedure is performing the augmentation by AHR 
5 Integration (step 4), which is achieved by applying a proper set of AH operators. 
Typical augmentations include: 

• Integration with virtual scene: 
A typical AR world normally consists of the objects constructed from the real 
3D scene and additional objects or backgrounds imported from the virtual 

10 world. One of the most important problems in the integration procedure Is the 

proper alignment of the virtual parts and the real parts. 

• Change of surface colors and/or textures: 
The objects in the AR world may change their surface colors and/or textures 
to present certain effects (for example, the shadow and the color changes 

rui5 due to the virtual light sources); 

• Shape changes by deformations: 
When the AR world is presented for physical simulation, it is possible that the 
objects, either from real views or from a virtual world, have shape changes 
due to the physical Interactions among them. 

p In Definition 12 and Definition 13. we have defined the operators u (union) and 

S n (intersection) on two attributed hypergraphs. which can be directly applied for 

the task of AHR Integration. Suppose that the AHR of the real scene is Grand 
the AHR of the virtual part is Gv, the combined AHR G is obtained by: 
25 1 . calculate Gu - Gr u Gv and Gi-Grr\ Gv, 

2. let Go = Gu \ G/; 

3. align the attributes of all entities in Gr. 

4. set G = Go u Gl. 



a 
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The alignment of the attributes of two AHR's depends upon the application 
context. Normally, it only involves translations and/or rotations on the AHR's to 
properly position the objects in the AR scene and thus is handled by one or more 
attribute transitions. Hovi^ever. In some cases, there may be qualitative changes 
5 of the AHR's associated with the integration. In the task of object fusion which 

may be required by AR tasks, the Integration of the two AHR's that represent the 
two objects, denoted as Grand Gv, has to be followed by eliminating all 
redundancies (e.g., inner surfaces or inner vertices) to ensure a fully integrated 
shape. The following steps will remove the Inner vertices and surfaces: 
10 1 . calculate G which Is the integration of Gr and Gv (with possible Inner vertices 
and/or surfaces), and G/ = Gm Gv; 

2. if a vertex vk e Gi, vk is an inner vertex: 

(a) find a vertex v/ e G such that vi Is adjacent to vk and the discrepancy 
between v^s attribute value Avi to vk's attribute value Avk Is the smallest 

IS ^5 among all vertices that are adjacent to vk; 

(b) in G. apply merge on vk and vi; 

3. if a hyperedge Hk e Gi, Hk becomes an Inner hyperedge: 

a) find a hyperedge H/ e G such that Hi is adjacent to Hk and the 
discrepancy between Hfs attribute value Aei to Hk's attribute value Aek is 

g20 the smallest among all hyperedges adjacent to Hk; 

b) InG, applyjoinonHfcandH/; 

4. If there is no inner vertex or hyperedge, exit, othen^vise go to step 2. 
Another important procedure in perfonning the augmentation Is object 

reconstruction from AHR. Since the AHR has the geometrical infomiatlon stored 
25 on Its elementary graph, which Is constructed from the triangular mesh model 
through RAG's, the reconstruction of the object shape resembles the 
reconstruction of an object's shape by triangular mesh approximations. Other 
properties of the object, such as surface color and texture can be extracted from 
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the attributes associated with the AHR and applied to the reconstructed 
triangular mesh. 

Natural looking surface textures (or surface colors when texture Is not 
considered) are of great importance to the construction of augmented reality. To 
5 meet this requirement, a texture or color or color binding algorithm is adopted to 
estimate the surface appearance for each triangle from the available image 
views. 

Figure 13 illustrates the principle of binding surface appearance information. 
With calibrated camera parameters, the projective relation between the 3D 
10 shape and their image views can be established in the form of a set of 

equations. Then for each triangular surface in the reconstructed model, the 
image view with the largest viewing area of the triangle Is selected. The texture 
and/or color of the projected triangle in the selected image view is inverted into 
Cj 3D modeling space and clipped on the corresponding triangle. The surface 

j y 1 5 texture (or color) binding algorithm has the following modules: 
5fl 1 • Texture (or color) region grouping: 

:| Since the object is represented by an AHR-based on triangular mesh. 

smoothly connected triangles with homogeneous textures (or colors) in their 
1=^ image views are grouped together to reduce the computational cost. Local 

|20 geometrical features are also considered as constraints to make sure that the 

triangles in the same texture (color) group are smoothly connected. A closed 
convex region that has invariant nomial vector is preferred for binding to 
reduce the total length of boundary. 
2. Image projection selection: 

In most cases, a grouped region is visible in more than one of the Images. It 
is reasonable to assume that the greater the exposure of the texture region In 
an image, the more accuracy will be achieved when the region is inverted into 
a model space. Therefore, the image with the largest viewing area of the 
grouped texture region is selected and a projective inversion is performed 
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with the camera (or any other sensor used for capturing the images) pose 
associated with this image. 

3. Boundary processing: 

Special care has to be taken along the boundaries between different texture 
regions. A simple averaging filter is added for the local areas around the 
boundaries to blend the texture. Although this results in a local blumng effect, 
it is more visually acceptable than the abmpt change without the filter, which 
is called the blocking effect in computer graphics. 

4. Texture synthesis for occluded region: 

It Is possible that the available views do not coverall the surfaces of the 
object/scene. For color attributes, a simple duplication of the color from the 
nearby triangles Is adequate. In the case that textures are processed, the 
texture of the invisible regions Is synthesized from visible regions depending 
on the viewing point. The synthesized textures or duplicated colors may not 
correspond to the real ones but are still less conspicuous then the 
texture/color holes. 



The problem of shape deformation arises when the objects/scenes are 
modeled In AHR with physical parameters as attribute values. Physical features 
as attributes In AHR describe deformable objects by (1) the physical properties 
of the objects; (2) the physical state transitions; (3) the initial states; and (4) 
external factors such as the external forces. 

In an augmented scene, the augmentation may include the changes of the 
physical states for the existing objects or the virtual objects, and possibly with 
additional external forces. In the above cases, due to the physical Interactions 
among the objects, there may be deformations. Therefore, a recalculation on the 
physical states of the objects is required. Further detail of the calculations on 
shape defomnations by AHR operators has been given in Section 8. 
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D. 3D Web Streaming based on AHR 

Referring again to Figure 1. the AHR method disclosed herein provides not 
only practical modeling and transformation methods but also an efficient data 
compression method for 3D data. A typical application of such data compression 
is 3D Web content streaming. AHR-based 3D Web streaming methodology 
utilizes the method given in Section A to model 3D Web content. It transmits the 
symbolic representation of the Web content instead of the Web content itself. 
Hence, it significantly improves the speed of content transmission over the 
Internet with limited bandwidth. The procedure of 3D Web streaming Is: 

1 . at the server side, given a piece of dynamic 3D Web content, extract the 
features and represent by AHR Gs; 

2. construct the AHR Gd corresponding to the dynamic portion of the Web 
content; 

3. extract the AHR operator sequence OPd as directed in Section B by 
comparing Gdwith its corresponding portion in Gs; 

4. transmit the AHR operator sequence OPd over the Internet; 

5. at the client side (on a client computer), given the AHR Gs and Its 
con-esponding operator sequence OPd, apply the operators to the AHR and 
generate a sequence AHR's, which represent the dynamic Web content; 

6. perform object reconstruction for each of the AHR to form the dynamic Web 
content. 

The foregoing description of the preferred embodiments of the Invention has 
been presented to illustrate the principles of the invention and not to limit the 
Invention to the particular embodiment Illustrated. It is intended that the scope of the 
invention be defined by all of the embodiments encompassed within the following 
claims and their equivalents. 
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